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Abstract. 

The thermodynamic and retrieval properties of fully connected Blume-Emery- 
Griffiths networks, storing ternary patterns, are studied using replica mean-field theory. 
Capacity-temperature phase diagrams are derived for several values of the pattern 
activity. It is found that the retrieval phase is the largest in comparison with other 
three-state neuron models. Furthermore, the meaning and stability of the so-called 
quadrupolar phase is discussed as a function of both the temperature and the pattern 
activity. Where appropriate, the results are compared with the diluted version of the 
model. 
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1. Introduction 

It has been argued recently that an optimal Hamiltonian, guaranteeing the best retrieval 
properties for neural networks with multistate neurons can be found by maximizing the 
mutual information content of the network [J. In this way, for two-state neurons 
the well-known Hopfield model is recovered || , for three-state neurons a Blume- Emery- 
Griffiths (BEG) type spin-glass model is obtained Q-0 (and references therein). In 
the discussion of the extremely diluted asymmetric version of this model Q, which 
has an exactly solvable dynamics because there are no feedback spin correlations ||, a 
new phase appeared, the so-called quadrupolar phase, that could yield new retrieval 
information. The time evolution of the order parameter characterizing this phase 
together with the stability properties of this phase in the extremely diluted model have 
been studied |§. Furthermore, the zero-temperature parallel dynamics of the fully 
connected architecture taking into account all feedback correlations has been solved 
recently [[Hj. No quadrupolar phase has been found in this case. 



A complete study of both the thermodynamic and retrieval properties of a fully 
connected architecture governed by such a BEG spin-glass Hamiltonian is not yet given 
in the literature. Although some preliminary results about the retrieval quality of this 
model for uniformly distributed patterns and non-zero temperatures have been presented 
in [0], one would like to have a complete temperature-capacity phase diagram as a 
function of the pattern activity. Furthermore, one would also like to solve the question 
posed in about the stability of the quadrupolar phase. Finally, one would like to know 
in more detail how the retrieval quality of this model compares with other three-state 
neuron models. To fill these gaps is the purpose of the present work. 

The main results obtained are the following. Using replica-symmetric mean-field 
theory it is shown that the retrieval phase is systematically larger than the one of the 
3-state neuron models known in the literature. The critical capacity of the BEG neural 



network is about two times bigger than the one of the 3-state neuron Ising model flT 
The region of thermodynamic stability of the retrieval states is much larger than the 
one for the 3-Ising model and, interestingly, even slightly bigger than the corresponding 
region for the Hopfield model. Next, it is found that the quadrupolar phase is not a stable 
solution for low temperatures but can become stable at high temperatures for suitable 
choices of the network parameters. The physical meaning of this is discussed. Finally, 
by calculating the zero-temperature entropy we expect that, for uniformly distributed 
patterns, replica-symmetry breaking is of the same order as the breaking in the Hopfield 
model. 

The rest of the paper is organised as follows. The model is introduced from a 
dynamic point of view in section £| Section ^ presents the replica-symmetric mean-field 
approximation and obtains the fixed-point equations for the relevant order parameters. 
In Section [| these equations are studied in detail for arbitrary temperatures. In 
particular, a temperature-activity phase diagram for low loading and temperature- 
capacity phase diagrams for finite loading and several pattern activities are obtained. 
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Also the specific thermodynamic properties are discussed. Where appropriate, the 
results are compared with the diluted version of the model. Concluding remarks are 
given in section [5|. Finally, the appendix contains the explicit form of the fixed-point 
equations. 



2. The model 

Consider a network of N neurons, {0";}, % = 1, . . . N, which take values out of the set 
{ — 1,0,1}. In this network we want to store p = aN patterns, {£f}, % = 1, . . . N 
and p, = 1, . . .p. They are supposed to be independent identically distributed random 
variables (iidrv) with respect to % and fi drawn from a probability distribution given by 

p(en = ^(i- (en 2 ) + w 

with a the pattern activity, viz. 

% 

The neurons are updated asynchronously according to the transition probability 
Pr {a[ = s e {-1, 0, l}|{a l} ) = ^tM^M (3 ) 

se{-i,o,i} 

with (3 the inverse temperature and ej(s|{o"j}) an effective single site energy function 
given by 0J 

e i (s\{a l }) = -h t ({a i })s-9 l ({a l })s 2 , se {-1,0,1} (4) 
where the random local fields are defined by 

N N 

k = J2 J ii a i » °i = E K a*j ■ ( 5 ) 
i=i 3=1 

The coefficients in these local fields are determined via the Hebb rule 

^ = ^tm, K^^^tvW, *=«*)' -a). (6) 

For zero temperature, the dynamical rule becomes 

a[ = BignMto})) 0(|/w(W)| + ei({tr t })) . (7) 
The long-time behaviour of this network is governed by the Hamiltonian |l|, |2|] 

H = ~\ E JuWi - \ E • (8) 

Since we want to compare this model with the 3-Ising model and we want to be able to 
change the relative importance of the two terms we rewrite the Hamiltonian as 

A B 

H = — 2 E kiwi - ~2 E KijOi°j , (9) 
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with 



Jjj = aJij , Kij = a(l — a)K, 



For 



B 



(10) 



11^ 



a a(l — a) 

we trivially recover the model above. When we now take = b5ij and A = B = 1 we 
obtain the 3-state Ising model | TT| . Finally, we find back the Hopfield model by taking 
first B = and then a = 1, again with A = 1. 



3. Replica-symmetric mean field theory 



We apply the standard replica technique |T2[ in order to calculate the free energy of 
the model. Within the replica-symmetry approximation and for a finite number, s, of 
condensed patterns, we obtain 

m = l±(aAml + o(l - a)B £) + |L log(l - x) + ^ log(l - <j>) 



2(3 



] a X _g g a #Pi< 



2/31-x 2/31 



2(1- X ) 2 2(1 



- - DsDt In Tr CT exp (/3# 
with the effective Hamiltonian H given by 



H = Aa 



ars 



Bo 7 



a Ax 



21-X 



■ a 



a 5, 
21^ 



a 



(12) 



(13) 



and where Ds and -Dt are Gaussian measures, Ds 
Furthermore 

<li 



X = Af3(q -q x ), = B(3(p - Pi) , r = _ , 
In these expressions the relevant order parameters are 



u 



^(2vr)- 1 / 2 exp(-s 2 /2). 
Pi 



(1 



a \ J /{£"} 

— - — - (V / DsDt (a 
a(l-a) V 7 \ 



9i = 
Pi = 



DsDt (a 



DsDt (of. 



(14) 

(15) 
(16) 
(17) 
(18) 
(19) 



DsDt (a 2 

where (■) ^ represents the thermal average with respect to the effective Hamiltonian H. 
In the sequel we take only one condensed pattern such that the index v can be dropped. 
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The parameter m is the usual overlap between the condensed pattern and the 
network state, while I is related to the activity overlap, i.e., (a 2 £ 2 ). In || it has 
been called the fluctuation overlap between the binary state variables of and rjf. 
Furthermore, go is the activity of the neurons and qi and pi are the Edwards-Anderson 
order parameters with their conjugate variables r, respectively u. Finally, \ and <fi 
are the susceptibilities proportional to the fluctuation of the m overlap, respectively I 
overlap. 

We remark that the trace over the neurons and the average over the patterns can 



be performed explicitly. The resulting expressions are written down in Appendix A 



4. Thermodynamic and retrieval properties 

In this section, we study the thermodynamic and retrieval properties of the fully 
connected BEG network by numerically solving the fixed-point equations (|l^)-(0) for 
one condensed pattern. Depending on the temperature T and on the system parameters 
a and a we recognize the following phases in terms of the order parameters. 

There is a retrieval phase R (m > 0, I > 0, q\ > 0) characterized by positive m 
and I and a quadrupolar phase Q (m = 0,/ > 0, gi = 0) where only I is non-zero, 
meaning that the active neurons (±1) coincide with the active patterns but the signs 
are not correlated. This implies that this phase also carries some retrieval information. 
From the fact that q\ = in this phase we know that the spins are not frozen, so 
that we expect to find this phase only for high enough temperatures. Furthermore, 
there is the spin-glass phase S (m = 0, 1 = 0, q\ > 0) and the paramagnetic phase P 
(m — 0, 1 = 0, qi = 0). We first look at low loading a = 0. 



4-1. Low loading 

For a = 0, the fixed-point equations simplify a lot because all the integrations (see 



Appendix A| ) drop out. This allows us to carefully study the quadrupolar state as a 



function of the pattern activity a since it turns out that the effect of the quadrupolar 
phase is strongest for a small loading capacity and high temperatures. A temperature- 
activity phase is presented in figure [H A dashed (full) line corresponds to a continuous 
(discontinuous) transition. 

Below a = 1/2, there is a continuous transition at glTr = 2/3 from the retrieval 
phase at low T to the paramagnetic phase at high T. This is similar to the 3-Ising 



model where it occurs for all values of a [ II]. At a = 1/2 the transition becomes 



discontinuous and, up to a = 0.698, the only phases present are R and P. The 
quadrupolar phase starts to appear at a = 0.698 and aT = 0.767, and beyond that 
point it keeps growing for increasing a. The transition R — Q remains discontinuous 
up to a = 0.708 and aT = 0.78 . For bigger values of a, it remains continuous and 
ends at aT = (1 — (2e) -1 ) -1 = 0.64 for a = 1. We remark that this phase diagram is 
the same, up to some slightly different numerical values for the transition points, as the 
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Figure 1. The BEG phase diagram for a — : aT as a function of a. Dashed (full) 
lines denote continuous (discontinuous) transitions. 



corresponding one for the extremely diluted model |§, confirming the fact that for low 
loading the architecture is not important. 

4-2. Finite loading 

Next, in figures 0, [3] and f|we present the temperature-capacity phase diagrams for three 
typical values of the pattern activity, a — 1/2, a — 2/3 (uniformly distributed patterns) 
and a = 0.8. As before, a dashed line corresponds to a continuous transition, while a 
full line corresponds to a discontinuous transition (in all order parameters). 

We start with some general observations. Below the line Tr retrieval states occur. 
The curve Tp represents the thermodynamic transition between retrieval states and 
spin-glass states. Hence, below Tp the retrieval states are global minima of the free 
energy while above this line the spin-glass states are. Thermodynamic transitions are 
shown as thick lines. The line T$g denotes the transition from the spin-glass to the 
paramagnetic phase. 

In all phase diagrams we notice some reentrance, i.e., a c at some finite T is larger 



than a c at T = 0. This effect is well-known from the literature (see, e.g., Jl3|], [Q) and 
signals the breaking of replica symmetry. In order to have an idea about the size of this 
breaking compared with other models, we calculate the entropy at zero temperature 



15 



ln(l - x) + ln(l - 4>) + 4 



(20) 



i-x i-< 

We find that this entropy is indeed negative but small. For uniform patterns, e.g., the 
entropy of the retrieval state at a c is, S(a c ) = —0.0017, which is of the same order of 
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magnitude as the one for the Hopfield model, i.e., S(a c ) = —0.0014, suggesting that the 
breaking is comparable. Moreover, the zero temperature entropy becomes more negative 
for increasing a, e.g., S(a c ) is —0.0010, —0,0017 and —0.031 for a = 1/2, a = 2/3 and 
a = 0.8 respectively, which might suggest that the breaking becomes larger. 

Next, we look at the different phase diagrams in more detail. In the case of a = 1/2 
shown in figure ^, the diagram quantitatively resembles the one for the 3-Ising model 
111]. At high temperatures there is the continuous transition from the disordered 




Figure 2. The BEG a — T phase diagram for a — 1/2. The meaning of the lines is 
explained in the text. 




Figure 3. The BEG a — T phase diagram for a = 2/3. The meaning of the lines is 
explained in the text 
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paramagnetic phase to the spin-glass phase. When crossing the curve Tr retrieval 
states show up as local minima of the free energy. At these points the overlap with 
the embedded patterns jumps from zero to a finite macroscopic value. In comparison 
with the 3-Ising model there is a small kink in the line Tr. When lowering T further 
the retrieval states become global minima of the free energy. This happens along the 
curve Tp and this thermodynamic transition is first order. The value for the critical 
capacity at zero temperature is a c = 0.058, while the thermodynamic transition point 
is a F = 0.030. 

For uniform patterns (see figure |3|) several transition curves bordering the different 
phases discussed above show up. Here we note that the critical curves Tsg and Tr end 
in different temperature points at a = giving rise to a 'crossover' region for small 



a as it occurs in the Potts model [[L6[]. This is related with the fact that for a = 
this model has a discontinuous transition at Tp. In this crossover region the retrieval 
states (global minima below Tp) and the paramagnetic states (local minima below Tp) 
coexist. Comparing these results with those found for the 3-Ising model we see 



that the a c = 0.091 found here is almost double of the critical capacity of the latter, 
a c = 0.046. We recall that the last number is obtained in the case of an optimal choice 
for the model parameter b, i.e. b = 1/2, which makes the Hamming distance minimal 



llf] . Furthermore, the region of thermodynamic stability for the retrieval states is about 



four times bigger. Compared with the Hopfield model, we notice that a c is smaller in 
the BEG model, 0.091 versus 0.13, but ap is larger, 0.053 versus 0.051. So a bigger 
number of the retrieval states in the BEG network are global minima of the free energy 
Figure ^ shows the phase diagram for a = 0.8. We immediately remark that the 
structure of the phase diagram turns out to be more complicated, as expected. The 




Figure 4. The a — T BEG phase diagram for a = .8. The meaning of the lines is 
explained in the text. 
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major difference with the foregoing phase diagrams is the presence of the quadrupolar 
phase for high temperatures. Indeed, from the discussion in section 4.1 (see figure we 
know that this quadrupolar phase shows up from an activity a = 0.698 onwards. As a 
consequence, the cross-over or coexistence region is larger than the one for the a = 2/3 
phase diagram. The transition from the Q phase to the P phase is discontinuous and by 
comparing the relevant free energies of the different phases we find the thermodynamic 
transition line Tp below which the quadrupolar states are global minima. We see that Tp 
joins nicely with Tp, as it should. The transition from the R to the Q phase is continuous 
for small a up to a = 0.0023 and discontinuous beyond that value. Finally, we notice 
that there is a larger reentrance suggesting a stronger replica-symmetry breaking, which 
is consistent with the fact that the zero-temperature entropy is more negative in this 
case, as mentioned above. 

The quadrupolar phase is situated in the high temperature region and we can 
understand the physics behind it in the following way. The spin-glass order parameter 
qi is zero, meaning that the ±1 spins are not frozen and as a consequence m can be zero. 
The fact that I is not zero practically means that the spins can flip freely between ±1 
but the probability that they jump to or vice versa becomes very small. This effect 
arises from a > 1/2 onwards when the ratio between the second and the first term in the 
Hamiltonian starts increasing as (1 — a) -1 . It implies that the information content of 
the system is non-zero in this phase. A practical example may be in pattern recognition 
where, looking at black and white pictures on a grey background, this phase would tell 
us the exact location of the picture with respect to the background without finding the 
details of the picture itself. 

Returning to figure |] we remark that in the shaded region two retrieval states 




Figure 5. The overlap to as a function of a for T — 0.2, 0.4, 0.6, 0.8. Full (dashed) 
lines denote stable (unstable) solutions. 
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coexist. A similar behaviour has also been seen in other multi-state networks, e.g., the 
fully connected Potts model Their presence can be understood by studying the 

overlap order parameter m as a function of a. This is shown in figure [| where the full 
line denotes stable solutions, while the dashed line corresponds to unstable solutions 
(saddle points). For small T two stable solutions occur. The one with the smallest 
overlap vanishes for large T. 

We end this section with a technical remark. In studying the fixed-point equations 
of this model, it turns out that a lot of solutions are in fact saddle points and not minima 
of the free energy. This can, of course, be investigated by studying the local stability of 
the extrema of the free energy. For large a we find, e.g., that there may be more than 
4 possible solutions involving some kind of quadrupolar character (m — 0, I > 0). Only 
the solution depicted in figure |] is a stable one (a real atractor). This also answers the 
question posed in about the stability of the quadrupolar state at low temperatures. 



4-3. Coefficients A and B versus the critical capacity 

In defining the BEG neural network model, we have included the possibility of varying 
the coefficients A and B in the Hamiltonian. The value of the coefficients A and B given 
by (|IT|) stems from [|], [|. They are obtained by optimizing the mutual information of 
the system. One could expect that this choice also improves other properties of the 
neural network, e.g., the basin of attraction, the critical capacity a c . 





Figure 6. The capacities a c and of as a function of 7 = a(l — a)B for uniform 
patterns at T — 0. 



Figure ^ shows a c (dashed line) and or (full line) as a function of 7 = a(l — a)B 
with A fixed at A = 1/a for uniform patterns at T = 0. For 7 = 1, we recover the 
BEG neural network as in (|TT|) and studied above. It turns out that the maximum in 
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the critical capacity is located at 7 = 0.712 with a corresponding value of a c = 0.096. 
Also the maximum in the thermodynamic transition is located at a value smaller than 
1. This does not agree with the expectation formulated above. 

A reason for this is the approximation done in order to get the mean-field 
Hamiltonian in 0. At a certain point in its derivation the authors assume that 
go ~ cl- Consequently, the mutual information of the network is optimized under this 
assumption. Although this assumption is natural for having a complete match between 
the final state of the network and the condensed pattern, in general, it may not be 
realized in a specific model. Furthermore, the fact that replica-symmetry breaking may 
be bigger for larger a, as is also suggested by the zero-temperature entropy calculation, 
could be an extra reason why the maximum in a c is further away from the value 7 = 1 
than is the maximum in otF- A further investigation is non-trivial and beyond the scope 
of the present work. 

5. Concluding remarks 

We have considered both the thermodynamic and retrieval properties of fully connected 
BEG networks. Fixed-point equations for the relevant order parameters have been 
derived for arbitrary temperatures in the replica-symmetric mean-field approximation. 
An activity-temperature phase diagram for low loading has been obtained. Near 
saturation, capacity-temperature phase diagrams have been discussed in detail for 
several values of the activity of the three-state patterns. 

Compared with existing three-state neuron models the retrieval region is larger 
and, e.g., for uniformly distributed patterns the critical capacity at zero temperature 
is almost two times bigger. Also the region of thermodynamic stability of the retrieval 
states is much enlarged and even larger than the one for the two-state Hopfield model. 
A new information carrying phase, the quadrupolar phase, appears at larger values of 
the activity in the high-temperature region of the phase diagram and may extend the 
practical usefulness of this network, e.g, in pattern recognition. 
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Appendix A. Explicit expressions for the fixed point equations 



In this appendix we write down explicitly the fixed-point equations for the order 
parameters of the BEG network. After performing the trace over the spins and the 
average over the condensed patterns in equations (p~5[)-(|T9t) we obtain 



m 



DsDt V 



/3 



DsDt W 



Am + Ay/ars) , \Bl(l - a) + By/out + ^Xj 
Am + Ay/ars^j , \Bl(l - a) + By/out + ^X^j 



DsDt W 



A\fars 



. , Bla + By/out + -X 
\ v 2 



q = Po = a J DsDt Wp (Am + Ay/ars) , (bI(1 - a) + By/out + ^X 

+(1 - a) J DsDt Wp (Ay/oUrs^ , (-Bla + By/out + ^. 
qi = a J DsDt (Vp [Am + Aa/ots) , ^B/(l - a) + By/aut + ^X 



1 \ 2 



+ (1 - a) y DsDt I V/9 
Pi = a J DsDt (Wp 

+(l-a) y .DsDt (w^ 

with 



a: 



-Sia + BJaut + -X 
v 2 



1 \ 2 



Am + Avars ) , ( 5/(1 - a) + By/aut + -X 



a 



l\2 



(Ay/ars) , ( -B/a + B^/aut + ^AT 



X = A(3(q -qx), 4> = B(3(po - p x ) , r 



qi 



u 



Pi 



(1 



and 



X = A- 



X 



B- 



l-X 1-0 
The functions and Wp are defined by 



2 exp(— /%) + cosh(/3a;) 



cosh(/3x) 



| exp(— /3y) + cosh(/3x) 



and reduce, for zero temperature, to 

Voo(x,y) = sign(x) Q(\x\ + y) 
Woo(ar,2/) = 0(|x| +y) . 



(A.l) 
(A.2) 
(A.3) 

(A.4) 

(A.5) 
(A.6) 
(A.7) 

(A.8) 
(A.9) 



(A.10) 
(All) 



